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(54) ELECTRONIC APPLIANCE 

(57)Abstract: 

PROBLEM TO BE SOLVED: To enable also a user 
inexperienced in the operation of a user interface expressed 
by an image projected on a projection surface to quickly 
recognize user's own misoperation. 

SOLUTION: The electronic appliance is a portable telephone 
set provided with: an image projection means 130 for 
projecting an interface image for expressing the user 'M 

interface to be operated by a user; a display control means r ^ ^ /_ . 



111 for controlling the display of the image projection means 
1 30; an operating position detection means 1 80 for 
detecting a user's operating position on the interface image; 
and a data processing means 111 for executing data 
processing corresponding to the user's operating position 
detected by the operating position detection means 180. 
The display control means 1 1 1 performs display control for 
changing at least a part of the interface image in 
accordance with the user's operating position detected by 
the operating position detection means 180. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 

[Field of the Invention] 

[0001] 

Based on the user's operation to the operation-sides picture which expresses virtually the operation 
sides of the operation equipment which a user operates, this invention, It is related with electronic 
devices, such as mobile telecom terminals, such as a portable telephone which performs 
predetermined data processing for control of various applications, such as an input of text, and a 
game, a sound, the output of a picture, etc., a touch panel, and an electric piano. 
[Background of the Invention] 
[0002] 

The projection means which projects the operation-sides picture which expresses the virtual 
keyboard for computers (operation equipment) virtually as this kind of an electronic device 
conventionally, The personal carrying type information terminal (PDA) provided with a detection 
means to detect which key was chosen on the virtual keyboard picture is known (for example, the 
patent documents 1 and 2 and three references). In this information terminal, a virtual keyboard 
picture is projected on an even and bright surface of projection, for example like white paper. A user 
is touching the keys with a finger or pointing by components, such as a pointer, and does key 
operation of the virtual key of the virtual keyboard projected on the surface of projection. This key 
operation is detected by the detection means which consists of laser sensors etc., and it can be 
inputted as if it touched the keys of the key on the physical keyboard for actual computers. 
[Patent documents 1] JP,2000-89899,A 
[Patent documents 2] US,6,614,422,B Description 
[Patent documents 3] JP,2003-152851,A 
[Description of the Invention] 
[Problem to be solved by the invention] 
[0003] 

In performing key operation to a keyboard, a user may touch accidentally the keys of between the 
key which he meant, and the next keys, or may touch the keys accidentally [ part / where the key is 
not arranged ]. In such a case, the keystroke which the user meant is not performed appropriately. 
When using a physical keyboard, since the error can be perceived with a finger, the user can notice 
that a keystroke was not appropriately performed from the feeling transmitted from a finger. 
However, when using a virtual keyboard, the user cannot take in the error with a finger, even when 
the touch which made the mistake in being such is performed. Therefore, the user cannot notice 
that a keystroke was not appropriately performed from the feeling transmitted from a finger. And 
when a user checks the operation result which he performed in this case in the pictures, having 
performed the touch which he mistook for the first time will be recognized. When it checks that the 
character etc. which were displayed by key operation did not specifically differ from what he meant, 
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or the character etc. which he meant had not been displayed in the pictures, it recognizes having 
performed the touch which he mistook for the first time. Since key operation can be performed 
looking at the screen where the result of key operation is displayed if it is the user who became 
skillful in operation of a keyboard, it can recognize promptly having performed the touch which he 
mistook. However, in order that a user unfamiliar to operation of a keyboard might perform key 
operation in many cases, looking at a keyboard, there was a problem that it was difficult to 
recognize promptly having performed the touch which he mistook. 

Although the above explained the case where the operation equipment virtually expressed by the 
picture projected on a surface of projection was a keyboard, the above-mentioned problem and the 
same problem may occur also about the case where they are other operation equipment, such as 
keyboards, such as a pointing device and a piano. 
[0004] 

in light of the above-mentioned problems, this invention comes out. Even if the purpose is a user 
unfamiliar to operation of the operation equipment virtually expressed by the picture projected 
upwards, it is providing the electronic device which can recognize the error of operation of it 
promptly. 

[Means for solving problem] 
[0005] 

To achieve the above objects, invention of Claim 1 , The image projection means for projecting the 
operation-sides picture which expresses virtually the operation sides of the operation equipment 
which a user operates, The display control means which performs display control of this image 
projection means, and an actuated-valve-position detection means to detect the user actuated 
valve position on this operation-sides picture, Are the data processing means which performs data 
processing corresponding to the user actuated valve position detected by this actuated-valve- 
position detection means the electronic device which it had, and the above-mentioned display 
control means, According to the user actuated valve position detected by the above-mentioned 
actuated-valve-position detection means, display control which changes at least a part of above- 
mentioned operation-sides picture is performed. 

In this electronic device, at least a part of operation-sides picture is changed according to the user 
actuated valve position detected by the actuated-valve-position detection means. Therefore, while 
a user looks at an operation-sides picture and operating it to this, it is recognizing the operation- 
sides picture having been changed visually, and it becomes possible to recognize whether operation 
which he mistook was carried out. For example, if at least a part of operation-sides picture is 
changed when a user operates the impossible position of user's operation accidentally to an 
operation-sides picture, the user who looked at this can be made to recognize the error. The thing 
which change into other pictures the picture currently displayed on the changed part and for which 
only the color and luminosity of the changed part are changed remarkably is also included in 
"change of an operation-sides picture" by a display control means. 

Invention of Claim 2 is characterized by the above-mentioned display control being what changes 
the image region of the user actuated valve position detected by the above-mentioned actuated- 
valve-position detection means in the electronic device of Claim 1. 

Since the image region of that user actuated valve position is changed when a user operates it to an 
operation-sides picture, it can be made to recognize in this electronic device whether the position 
which he meant to the user who looked at this has been operated. 

Invention of Claim 3 is characterized by the above-mentioned operation-sides picture being a 
picture of the keyboard side where two or more keys for performing mutually different data 
processing to the above-mentioned data processing means were arranged in Claim 1 or the 
electronic device of 2. 

In this electronic device, a user can perform the same operation as the time of connecting this, 
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without connecting a physical keyboard. 
[0006] 

The operation-sides picture projected by the image projection means may be a two-dimensional 
picture projected on a surface of projection, and may be a three-dimensional picture projected on 
three-dimensional space by the holographic technique etc. which are recognized visually so that 
operation equipment may exist in three-dimensional space. 

Above a "electronic device", a personal digital assistant (PDA:Personal Digital Assistance), a game 
machine besides a mobile telecom terminal, etc. are contained. Above a "mobile telecom terminal", a 
PDC (Personal Digital Cellular) system, A GSM (Global System for Mobile Communication) system, 
Portable telephones, such as a TIA (Telecommunications Industry Association) system, and the 
portable telephone standardized by IMT(International Mobile Telecommunications)-2000 are 
contained. The mobile telecom terminal which added the cellular-phone module besides telephones, 
such as PHS (Personal Handyphone Service) and a car telephone, is also contained in a "mobile 
telecom terminal." 

The control in the above-mentioned electronic device is also realizable by executing a 
predetermined program by computer formed in them. Delivery of a program used by this computer 
may be performed using recording media which recorded the program as digital information, such as 
FD and CD-ROM, and may be performed using communication networks, such as a computer 
network. 

[Effect of the Invention] 
[0007] 

According to invention of Claims 1-3, by recognizing visually that the operation-sides picture was 
changed. Since it becomes possible to recognize whether operation which he mistook was carried 
out, even if it is an unfamiliar user who operates the operation equipment which this picture 
expresses virtually, looking at that operation-sides picture, the outstanding effect of becoming 
possible to recognize the error of operation of one promptly is done so. As a result, it becomes 
possible to raise the convenience of the user in the case of using the virtual operation equipment 
expressed by the operation-sides picture. 

Since it can be made to recognize especially whether the position which he meant to the unfamiliar 

user has been operated exactly according to invention of Claim 2, while a user checks an operation 

result, the outstanding effect that exact operation can be performed now is done so. 

According to invention of Claim 3, the outstanding effect that miniaturization of the electronic 

device concerned, portable improvement, etc. can be aimed at is done so. 

[Best Mode of Carrying Out the Invention] 

[0008] 

Hereafter, an embodiment of the invention is described, referring to Drawings. 

First, the entire configuration of the portable telephone which is a mobile telecom terminal as an 

electronic device concerning this embodiment is explained. 

Drawing 2 is a block diagram showing an example of the outline composition of the portable 
telephone concerning this embodiment. The body part 100 of this portable telephone is provided 
with the Radio Communications Department 113 as the main control part 111 as a control means, 
the internal memory 112 as a memory measure, and a means of communication, and the antenna 
114. The microphone 116 and the loudspeaker 1 17 are connected to the main control part 111 via 
the voice processing part 1 1 5, and the picture display part 1 1 9 is connected to it via the image 
processing portion 118, and also the key operation section 120 provided in the portable telephone 
body is connected to it. 

The portable telephone of this embodiment is provided with the following. 
The picture projection part 130 as an image projection means for projecting a picture on the 
external surface of projection (only henceforth a "surface of projection") 200 which a user can 
recognize visually. 
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The camera part 140 as an imaging means which picturizes pictures, such as a person and scenery. 
The portable telephone is provided with the key selection detection part 180 as an actuated-valve- 
position detection means by which a user detects the user actuated valve position on the virtual 
keyboard picture (operation-sides picture) projected on the operational surface of projection 200. 
By this key selection detection part 180, it is detectable which key keyboard picture up [ virtual ] 
was chosen. The detailed composition and operation of the picture projection part 130 and the key 
selection detection part 180 are mentioned later. 
[0009] 

The above-mentioned main control part 111 comprises a CPU, cache memory, a system bath, etc., 
for example, transmits and receives data or controls each part by executing a predetermined control 
program between each part of the internal memory 112 or Radio Communications Department 113 
grade. The main control part 1 1 1 functions also as a display control means which performs display 
control of the projection picture (keyboard picture) by the picture projection part 1 30. The main 
control part 1 1 1 is used also for control of the key selection detection by the key selection 
detection part 180. The main control part 111 is used based on the key select data outputted from 
the key selection detection part 180 also as a data processing means which performs predetermined 
data processing corresponding to the selected key. This key select data is data in which the user 
actuated valve position (key) on the keyboard picture projected by the picture projection part 130 is 
shown. Data processing which displays the alphabetic data corresponding to the key selected, for 
example on a display as the above-mentioned data processing, or is memorized to an internal 
memory, and data processing to which the contents of execution of application programs, such as a 
game, are changed are mentioned. 
[0010] 

The above-mentioned internal memory 112 memorizes the control program and various data which 
semiconductor memory, such as RAM and ROM, is consisted of, for example, and are performed by 
the main control part 111. This internal memory 112 is used also as a contents data storage means 
which memorizes the contents data of a picture, music, a program, etc. downloaded from the offer- 
of-information site etc. This internal memory 1 1 2 is used also as a data storage means which 
memorizes the data of the sound outputted from the loudspeaker 117, the image data displayed on 
the picture display part 119, the image data projected by the picture projection part 130, and the 
key select data detected by the key selection detection part 1 80. 
[0011] 

The above-mentioned Radio Communications Department 1 1 3 is controlled by the main control part 
111, and performs radio between the base stations of the portable telephone communication 
network as a communication network with a predetermined communication method via the antenna 
114. By this radio, voice Telephonic Communications Division can be performed between specific 
portable telephones etc., or data communications, such as transmission and reception of an E-mail 
and contents download from an offer— of-information site, can be performed. 
[0012] 

The above-mentioned voice processing part 1 1 5 codes the transmission voice signal inputted from 
the microphone 1 16 by a predetermined system, and sends it to the main control part 111. This 
voice processing part 115 decrypts the receiver voice signal received in the Radio Communications 
Department 113, and outputs it from the loudspeaker 117. This voice processing part 115 outputs 
correspondence items, such as a sound effect used with the ringtone memorized by the internal 
memory 112 or an application program, from the loudspeaker 117. The sound output means which 
outputs a sound is constituted using the voice processing part 115 and the loudspeaker 117. 
[0013] 

The described image treating part 118 processes image data, such as image data which received in 
the Radio Communications Department 113, an icon memorized by the internal memory 112, a menu, 
and an incoming call notice picture, and is made to display it on the picture display part 1 1 9 which 
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consists of liquid crystal displays (LCD) etc. 
[0014] 

The above-mentioned key operation section 1 20 is provided with a data input key (a ten key, the * 
key, the # key), a call start key, a clear back key, a scroll key, a multi function key, etc., and uses 
them for scrolling, selection, etc. of information which are displayed on the picture display part 119 
besides dispatch of a telephone, or arrival. This key operation section 120 is used also as a setting 
means for specifying at least one of two or more of the key selection detection functions which the 
key selection detection part 180 has. 
[0015] 

Drawing 3 is a block diagram showing the example of 1 composition of the described image 
projection part 130. 

This picture projection part 130 is constituted using the drive control section 133 for driving the 
beam light source 131, the optical scanner 132, and them. As the beam light source 131, LD 
(semiconductor laser), LED (light emitting diode), etc. can be used. The intensity of the beam 
emitted from the beam light source 131 is controlled by the drive control section 133 based on the 
control data sent from the main control part 111. The optical scanner 132 is Optical Apparatus Sub- 
Division projected on the surface of projection 200, scanning the beam emitted from the beam light 
source 131 in two dimensions. Drive controlling of the scan of the beam by the optical scanner 132 
is carried out by the drive control section 133 like the beam light source 131. As this optical 
scanner 132, the thing using the galvanomirror in which it is small and rapid scanning is possible can 
be used, for example. The above-mentioned drive control section 133 has an image memory which 
memorizes image data, and the image data of the projecting object by which registration of the 
establishment was carried out beforehand, and the image data of the projecting object sent with the 
control data from the main control part 111 are memorized. Based on the image data memorized by 
this image memory, drive controlling of the above-mentioned beam light source 131 and the optical 
scanner 132 is carried out. If each beam is scanned simultaneously and projected using what can 
emit the beam (for example, three-primary-colors beam) which is two or more kinds from which 
wavelength differs mutually as the above-mentioned beam light source 131, a color picture can be 
projected on the surface of projection 200. The light source which consists of a one dimensional 
array element with which the emission point in which intensity control is possible was located in a 
line in the shape of one row mutually-independent can also be used as the above-mentioned beam 
light source 131. In this case, what is necessary is just to scan the light of the line form emitted 
from the light source in one dimension with the optical scanner 132. The light source which consists 
of a two-dimensional array element with which the emission point in which intensity control is 
possible was located in a line in two dimensions mutually-independent can also be used as the 
above-mentioned beam light source 131. In this case, the extended projection optical system which 
expands the two-dimensional picture displayed on the light source instead of the optical scanner 
132, and is projected on the surface of projection 200 is used. 
[0016] 

Drawing 4 is a block diagram showing other examples of composition of the described image 
projection part 130. 

This picture projection part 130 is constituted using the light sources 134, such as a lamp, the 
spatial-light-modulation machine 135, the extended projection optical system 136, and the drive 
control section 133. As the spatial-light-modulation machine 135, the liquid crystal display currently 
used, for example with the usual portable telephone etc. can be used. In this case, a picture is 
displayed in white contrary to the display mode in a transmission type liquid crystal display, and it 
displays. It is displayed in white in this way, the surface of the displayed liquid crystal display is 
irradiated with the light from the light source 134, the reflected figure is expanded by the extended 
projection optical system 136, and it projects on the surface of projection 200. As this liquid crystal 
display, the liquid crystal display provided with the usual portable telephone etc. may be made to 
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serve a double purpose. As the spatial-light-modulation machine 1 35, what put the controllable 
small mirror in order for inclination in two dimensions mutually-independent can also be used. In this 
case, based on image data, inclination of each mirror is controlled and the reflected light of the light 
which goes to the surface-of-projection 200 side from each mirror is turned on and off. The 
reflected figure from this spatiaHight-modulation machine 135 is expanded by the extended 
projection optical system 136, and is projected on the surface of projection 200. 
[0017] 

Drawing 5 is an internal configuration figure of the portable telephone provided with the picture 
projection part concerning the example of composition of further others. 
The picture projection part 130 of this example of composition is constituted using optical 
generation part 130' for generating the light figure of a projecting object, and the projection imaging 
optical system for carrying out image formation of the light figure generated by optical generation 
part 130' on the surface of projection 200, and projecting it. As this optical generation part 130', 
below-mentioned LED section 137 and the liquid crystal panel part 138 are made to serve a double 
purpose, and the below-mentioned optical system 150 for camera image formation is made to serve 
a double purpose as the above-mentioned projection imaging optical system. The light emitted from 
LED section 137 is modulated by the liquid crystal panel part 138 controlled based on image data, 
and, thereby, the light figure of a projecting object is generated. The light figure of this projecting 
object is projected on the surface of projection 200 by the optical system 150 for camera image 
formation made to serve a double purpose as an optical system for projection imaging. 
The camera function in this example of composition is realized by the camera part 140 which 
changes a light figure into an electrical signal and generates image data, and the optical system 150 
for camera image formation for carrying out image formation of the light figure for an image pick-up 
on the camera part 140. The camera part 140 comprises a CMOS type imaging device, a CCD type 
imaging device, etc., and the drive of these imaging devices, the picturized preservation of image 
data, etc. are controlled by the main control part 111. The optical system 150 for camera image 
formation has the drive mechanism to which the position of two or more lenses is changed so that a 
zoom function may be given, and this drive mechanism is controlled from the main control part 111. 
Image formation of the light figure for an image pick-up is carried out on the acceptance surface of 
the camera part 140 by the optical system 150 for camera image formation. This light figure by 
which image formation was carried out is changed into an electrical signal by the camera part 140, 
and image data is generated. The image data generated by the camera part 140 is saved at the data 
storage part of internal-memory 1 1 2 grade. 

The lighting lamp function in this example of composition is realized by the optical generation part 
130 which consists of LED section 137 as a light source, and the liquid crystal panel part 138 as a 
spatial optical modulation part, and the optical system 155 for diffused irradiation for diffusing the 
light emitted from that light source, and carrying out a uniform exposure in an external irradiation 
surface. LED section 137 is constituted by the white bright luminescence diode, for example, and 
ON/OFF of luminescence, etc. are controlled by the main control part 111. The liquid crystal panel 
part 138 is constituted so that the transmissivity of the pixel of a large number formed in two 
dimensions can be controlled mutually-independent based on the control data from the main control 
part 111. In this example of composition, although the transmission type liquid crystal panel is used, 
a reflection type liquid crystal panel may be used. The light emitted from LED section 137 is 
uniformly penetrated by the liquid crystal panel part 138 controlled so that light transmittance 
became the maximum about an entire pixel, and the light for diffused irradiation is generated. The 
light for [ which was generated by this liquid crystal panel part 1 38 ] diffused irradiation makes it 
spread in the optical system 155 for diffused irradiation, and a uniform exposure is carried out in an 
external irradiation surface. 
[0018] 

In order to switch and use the above-mentioned camera function and a lighting lamp function, and 
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the function of described image projection, it comprises an example of composition shown in drawing 
5_so that a position may be interlocked with and the camera part 1 40 and optical generation part 
130' (LED section 137 and the liquid crystal panel part 138) can be moved. 

As shown in drawing 5 (a), when using a camera function and a lighting lamp function, the camera 
part 140 and optical generation part 130' (LED section 137 and the liquid crystal panel part 138) are 
moved to the position of the following. That is, it is made to move to the position A for diffused 
irradiation which turns optical generation part 130' (LED section 137 and the liquid crystal panel part 
138) to the optical system 155 for diffused irradiation, and emits light. The camera part 140 is 
moved to the position B for image formation which receives the light figure from the optical system 
150 for camera image formation. On the other hand, as shown in drawing 5 (b), when using the 
function of image projection, the camera part 140 and optical generation part 130' (LED section 137 
and the liquid crystal panel part 138) are moved to the position of the following in accordance with 
the direction shown by the black-lacquered arrow in a figure. That is, optical generation part 
130' (LED section 137 and the liquid crystal panel part 138) is replaced with the camera part 140, 
and is moved to the position B for image formation. The camera part 140 is moved to the retreating 
position C evacuated from the position B for image formation. It is in the state to which it was made 
to move in this way, and the opening which was awaited, for example and the optical system 150 for 
camera image formation exposed in the state is placed on a desk towards the desired surface of 
projection 200, or it sets to the battery charger for portable telephones. A user sets it as either of 
the image projection modes which performs image projection by operating the key operation section 
120. 
[0019] 

As shown in drawing 6 (a), when the picture display part 119 of the portable telephone comprises 
two liquid crystal displays (the transverse-plane display 119a and the back display 119b), one back 
display 1 19a may be made to serve a double purpose as a light source of a picture projection part. 
In this case, the picture displayed on the back display 1 1 9a is projected by the extended projection 
optical system 136 on the surface of projection 200. This extended projection optical system 136 is 
constituted removable to the portable telephone body 100. Usually, it is removed from the portable 
telephone body 100, and when using it for image projection, it can equip on the back display 1 19b of 
the portable telephone body 100 by the wearing holding part 136a provided in both ends. When 
projecting the picture of the back display 1 19b like drawing 6 (b), it is preferred to control by the 
main control part 1 1 1 to raise the light volume of the back display 1 1 9b rather than the time of the 
usual image display. In this case, the visibility of the projected picture can be raised. In the 
composition of drawing 6 (a) and (b), the projection optical system 136 which carries out the flip 
horizontal of the picture of the back display 1 19b, and projects it may be used. In this case, at the 
time of image projection, a flip horizontal is carried out to the time of the usual image display, and a 
picture is displayed on the back display 1 1 9b as it. Thereby, pictures, such as text, can be projected 
for [ as the time of the usual image display / same ] the right-and-left rights. 
[0020] 

The composition of the picture projection part 130 is not limited to the composition of drawing 3 - 
drawing 6 . For example, it may be the composition which is projected using a hologram element. 
[0021] 

Drawing 7 is the explanatory view in which the user showed signs that the virtual keyboard picture 
210 was projected on the predetermined projection area on an operational surface of projection, in 
the position which is separated from the main part of the foldaway portable telephone 100 provided 
with the described image projection part 130. This portable telephone 100 places the optical 
emitting port of the picture projection part 130 on a desk towards a desired surface of projection, or 
sets it to the battery charger for portable telephones. If virtual key operation mode is chosen and 
performed when a user operates the key operation section 120, the virtual predetermined keyboard 
picture 210 which the user specified beforehand will be projected on a projection area. It is detected 
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by the key selection detection part 180 which key on the keyboard picture 210 projected on the 

projection area by this picture projection part 130 was chosen. 

[0022] 

Next, the key selection detection part 180 in the portable telephone of this embodiment is explained. 

Drawing 8 is a block diagram showing the outline composition of the example of 1 composition of the 
above-mentioned key selection detection part 180. 

This key selection detection part 180 is constituted using the keystroke operation detecting part 
160 as a keystroke operation detection means, and the key select data generation part 170 as a key 
select data creating means. The keystroke operation detecting part 1 60 is what detects keystroke 
operation of key-input-operations things, such as a finger for the key selection on the virtual 
keyboard picture 210, and a pointer, It is constituted using two the optical outgoing radiation and the 
detection parts 161,162 provided in the both ends near the hinge region of the body part 100a of the 
portable telephone 100, and light source actuators 163. The key select data generation part 170 
generates the key select data corresponding to the keystroke operation by a key-input-operations 
thing based on the detection result of the keystroke operation detecting part 1 60. 
[0023] 

When inputting by projecting the virtual keyboard picture 210 with the portable telephone of this 
embodiment, as shown in drawing 7 , the covering device 100b is rotated 180 degrees like the arrow 
B centering on the axis of rotation A, and a portable telephone is installed. While a user performs 
key input operations on the virtual keyboard picture 210 by installing in this way, pictures, such as 
text displayed on the picture display part (liquid crystal display) 119 and a game under execution, 
can be checked. 
[0024] 

Each optical outgoing radiation and detection part 161,162 are constituted by the laser light sources 
161a and 162a which consist of a semiconductor laser etc. which emit a laser beam, and the photo 
sensors 161b and 162b which consist of a photo-transistor, a CCD array element, etc. The laser 
light sources 161a and 162a scan and emit a laser beam to a flabellate form so that the whole 
keyboard picture 210 on the projection area may be covered along the field of the projection area 
where the keyboard picture 210 is projected, respectively. The photo sensors 161b and 162b receive 
the reflected laser beam which the laser beam from each laser light sources 161a and 162a was 
interrupted by the user's finger 300 and pointer component, and has been reflected, respectively, 
and output a detection signal. In the composition of drawing 8 , the two above-mentioned photo 
sensors 161b and 162b need to receive only the laser beam from the laser light source which is a 
group, respectively, but there is a possibility of receiving light simultaneously and carrying out 
erroneous detection of the reflected laser beam emitted and reflected from both laser light sources 
161a and 162a. For example, erroneous detection of the reflected laser beam emitted and reflected 
from the laser light source 161a may be received and carried out not with the photo sensor 161b 
which is a group but with another photo sensor 162b. Then, in order to prevent the above- 
mentioned erroneous detection, it is preferred to change mutually the wavelength of the laser beam 
used by each optical outgoing radiation and detection part 161,162. While changing the wavelength 
of a laser beam instead of changing the wavelength of a laser beam, it may be made to modulate the 
laser beam used by each optical outgoing radiation and detection part 161,162 in mutually different 
code. In this case, in each optical outgoing radiation and detection part 161,162, only the reflected 
laser beam modulated in predetermined code, respectively can be received, recovery processing can 
be carried out, and it can output as a detection signal. 
[0025] 

The above-mentioned light source actuator 163 is controlled according to the control command 
from the above-mentioned main control part 111, impresses predetermined driver voltage to the 
above-mentioned laser light sources 161a and 162a and the photo sensors 161b and 162b, or 
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supplies driving current. The light source actuator 163 changes the degree of laser beam emitting 
angle emitted from the laser light sources 161a and 162a according to the control command from 
the main control part 111. 
[0026] 

The above-mentioned key select data generation part 1 70 is controlled according to the control 
command from the above-mentioned main control part 111, The key select data in which it is shown 
which key was chosen is generated by calculating using each degree theta 1 of laser beam emitting 
angle when a reflected laser beam is detected, and the data of theta 2 with the above-mentioned 
photo sensors 161b and 162b. This operation is performed using the predetermined data table for an 
operation and operation algorithm corresponding to the kind of keyboard picture which is projecting 
[ above-mentioned ]. This key select data generation part 170 may be formed in the main control 
part 1 1 1 , and it may be constituted so that the main control part 1 1 1 may be made to serve a 
double purpose. 
[0027] 

Drawing 9 is a partial side view of the portable telephone which has projected the keyboard picture 
210. Drawing 10 is an explanatory view explaining the principle of the key selection detection in the 
above-mentioned key selection detection part 180. 

If it is made to move to the position of the key (for example, "G" of the alphabet) of the selection 
object shown with the mark 210a and the finger 300, a pointer, etc. are chosen in drawing 9 and 
drawing 10 , Laser beam Li 1 emitted by scanning an angle from the laser light sources 161a and 162a 
of each optical outgoing radiation and detection part 161,162 and Li2 are interrupted and reflected. 
This laser beam Li1 and Li2 are reflected when set to the predetermined degree theta 1 of emitting 
angle, and theta 2, respectively. Each reflected laser beam Lr1 and Lr2 are received with the 
corresponding photo sensors 161b and 162b, respectively. The degree theta 1 of emitting angle of 
the laser light sources 161a and 162a when each reflected laser beam Lr1 and Lr2 are detected with 
these photo sensors 161b and 162b, and the data of theta 2 are sent to the above-mentioned key 
select data generation part 1 70, and are used for generation of key select data. Namely, since it will 
be uniquely decided if the degree theta 1 of emitting angle of the above-mentioned laser light 
sources 161a and 162a and the value of theta 2 are known, selected position (position on which the 
picture of the "G" key in drawing 1 0 w as projected), i.e., user actuated valve position, of a key, The 
key select data which identifies the selected "G" key is generable. 

Also when two keys and three keys or more are chosen simultaneously, by the same principle, 
selection of each key can be detected and two or more key select data can be generated. 
[0028] 

The main control part 1 1 1 performs predetermined data processing corresponding to the selected 
key based on the key select data outputted from the key select data generation part 1 70 of the 
above-mentioned key selection detection part 180. For example, the alphabetic data corresponding 
to the selected key is displayed on the picture display part (liquid crystal display) 119, or is 
memorized to an internal memory. Based on the above-mentioned key select data, the contents of 
execution of application programs, such as a game, are changed. 
[0029] 

When generating key select data from the above-mentioned degree theta 1 of emitting angle, and 
the value of theta 2, what was beforehand set up by the relation with the keyboard picture 210 
projected by the described image projection part 130 is used for the data table for an operation. Or 
before performing the keystroke using the above-mentioned keyboard picture 210, the degree of 
emitting angle is measured about the predetermined standard key (the key of four corners, and a 
central key) on the keyboard picture, The data table for an operation is amended using the value, 
and it may be made to generate more correctly in quest of the above-mentioned key select data. 
[0030] 

Although the degree theta 1 of emitting angle in each optical outgoing radiation and detection part 
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161,162 and the value of theta 2 are calculated, it may be made to calculate the distance D1 of 
each optical outgoing radiation and detection part 161,162, and the key of a selection object, and 
the value of D2 in the example of above-mentioned drawing 10 , as shown in drawing 1 1 . This 
distance D1 and the value of D2 can be calculated from the grade of attenuation of each reflected 
laser beam Lr1 which receives light with the photo sensors 161b and 162b of each optical outgoing 
radiation and detection part 161,162, and Lr2, for example. The above-mentioned distance D1 and 
the value of D2 may be calculated from interference with outgoing radiation laser beam Li 1 , Li2, and 
reflected laser beam Lr1 [ corresponding, respectively ] and Lr2. 
[0031] 

Although optical outgoing radiation and the detection part 161,162 are formed in two places in the 
example of drawing 7 - drawing 1 1 , it provides in three places or four places or more, and may be 
made to detect the above-mentioned key selection more correctly. 

When a user's finger is in a home position and the laser beam from above-mentioned optical 
outgoing radiation and detection part 161,162 is interrupted, change of the euphotic intensity of the 
reflected laser beam accompanying up-and-down motion of a finger may be used on the key image 
of a selection object. For example, when a user moves a finger up and down on the key image of a 
selection object, the intensity of each reflected laser beam Lr1 which receives light with the photo 
sensors 161b and 162b, and Lr2 is set to a zero level only fixed time. When there is change of the 
intensity of this each reflected laser beam Lr1 and Lr2, it judges with that key having been chosen 
and key select data is generated. 
[0032] 

Next, the display control which changes the keyboard picture 210 according to the user actuated 
valve position which is a characterizing portion of this invention is explained. 
According to this embodiment, when a user chooses a key, the color of the key image is changed 
and selection key picture change processing which makes it recognize whether the key which he 
meant to the user has been operated is performed. 

Drawing 1 is a control block diagram about the selection key picture change processing in this 
embodiment. 

As mentioned above, when a user performs key operation using the finger 300 as opposed to the 
virtual keyboard picture 210, the finger 300 is detected by the key selection detection part 180, and 
the key select data is sent to the main control part 111. Then, the main control part 1 1 1 sends the 
selection key color variation order for changing the color of the key corresponding to the key select 
data to the picture projection part 130 while performing predetermined data processing 
corresponding to the key select data. The picture projection part 1 30 which received this changes 
the parameter of the color of the image region of the key concerning a selection key color variation 
order about the image data of the keyboard picture 210 projected now according to the command. 
And the keyboard picture 210 based on the image data after change is projected on the surface of 
projection 200. 
[0033] 

By performing such processing, as shown in drawing 12 , the color of the key C corresponding to the 
user actuated valve position which the user operated is changed. Therefore, the user can recognize 
whether operation of it was received exactly by change of the color. 

The user can recognize whether the key which he meant has been operated exactly by change of 
the color. For example, as shown in drawing 13 , the user tried to operate the key C, but it is 
assumed that around between the key C and the keys D under it has been operated accidentally. In 
this case, it has not recognized which key was conventionally received by that key operation until it 
checked the character etc. which were inputted by that key operation in the pictures. On the other 
hand, since the color of the received key is changed immediately according to this embodiment, in a 
graphic display, the user who is operating it while looking at the keyboard picture 210 can recognize 
immediately that the key D was received. Therefore, although the user tried to operate the key C, 
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he can notice that it has operated the key D under it accidentally immediately. 
[0034] 

As mentioned above, since operation in which a user projects the virtual keyboard picture 210 and 
chooses a key on the keyboard picture on an operational field can be performed according to this 
embodiment, without connecting separate members, such as a keyboard, the portability of a portable 
telephone is securable. And when a user performs operation which chooses a key on the keyboard 
picture 210 according to this embodiment, the color of the picture of the key corresponding to the 
user actuated valve position, i.e., the picture of the key received by the portable telephone, is 
changed. Therefore, even if a user does not check the character etc. which were displayed as an 
operation result of the received key by the picture display part 119, he is only looking at the 
keyboard picture 210 which he is operating, and can check the received key. Therefore, even if it is 
a user unfamiliar to keyboard grabbing which performs key operation, looking at a keyboard, when 
mistaken operation has been carried out, you can notice the error immediately. 
[0035] 

[Modification(s)] 

Next, the modification of the portable telephone concerning the above-mentioned embodiment is 
explained. 

Although the above-mentioned embodiment explained the case where the operation-sides apparatus 
virtually expressed with the projection picture by the picture projection part 130 was a keyboard, 
this invention is applicable to various operation equipment. In this modification, the case where it 
applies to the operation equipment for handwritten inputs is mentioned as an example before long. 
[0036] 

Drawing 14 is a block diagram showing the outline composition of the example of 1 composition of 
the actuated-valve-position detection part 380. 

This actuated-valve-position detection part 380 is the same as that of the above-mentioned key 
selection detection part 1 80 except replacing with the key select data generation part 1 70 in the 
above-mentioned embodiment, and having the actuated-valve-position data generating part 370. 
The actuated-valve-position data generating part 370 is based on the detection result of each 
optical outgoing radiation and detection part 161,162, generates continuously the actuated-valve- 
position data in which the user actuated valve position on the handwritten input operation-sides 
picture 310 is shown, and outputs this to the main control part 111 continuously. Thereby, the main 
control part 1 1 1 can grasp continuously the actuated valve position on the handwritten input 
operation-sides picture 310 which the user's finger 300 operated. That is, the main control part 1 1 1 
can grasp the locus of a user actuated valve position from the actuated-valve-position data 
continuously outputted from the actuated-valve-position data generating part 370. And the main 
control part 1 1 1 performs data processing, such as character recognition processing which 
recognizes as a character the character which the user drew with the finger 300 on the handwritten 
input operation-sides picture 310, a picture, etc., and processing recognized as a picture. 
[0037] 

The main control part 1 1 1 performs input locus image change processing which changes the color of 
the picture of locus portions, such as the character, when a user draws a character etc. with the 
finger 300 to the handwritten input operation-sides picture 310. 

If it explains concretely, the main control part 1 1 1 will send the actuated-valve-position color 
variation order for changing the color of the picture of the actuated valve position corresponding to 
the actuated-valve-position data to the picture projection part 1 30, if the actuated-valve-position 
data from the actuated-valve-position data generating part 370 is received. The picture projection 
part 130 which received this changes the parameter of the color of the image region of the actuated 
valve position concerning the actuated-valve-position color variation order like the case of the 
above-mentioned embodiment according to the command about the image data of the handwritten 
input operation-sides picture 310 projected now. And the keyboard picture 210 based on the image 
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data after change is projected on the surface of projection 200. 
[0038] 

By repeating such processing, whenever it receives the actuated-valve-position data from the 
actuated-valve-position data generating part 370, as shown in drawing 1 5 , the color of the position 
corresponding to the user actuated valve position which the user operated is changed. Therefore, 
the user can recognize the position which he has operated by change of the color, and can grasp 
the character which he is drawing. According to this modification, since the user can check the 
character which he is drawing, looking at the handwritten input operation-sides picture 310 which is 
an object which he operates, when the character whose intention he has cannot be being drawn 
exactly, he can notice this immediately. 
[0039] 

In the above-mentioned embodiment and the above-mentioned modification, when a user operated 
it, the case where the color of the picture corresponding to the user actuated valve position was 
changed was explained, but the luminosity of a picture may be changed or it may change into 
another picture. 

In the above-mentioned embodiment and the above-mentioned modification, when the operation 
equipment virtually expressed with the projection picture by the picture projection part 1 30 was a 
keyboard, the case where it was operation equipment for handwritten inputs was explained, but in 
addition to this, this invention is applicable to various operation equipment. For example, it is 
applicable also to operation equipment like the keyboard of a piano. As shown in drawing 1 6 , the 
picture of a menu screen can be projected by the picture projection part 130, and it can also use as 
operation equipment like the touch panel for selection of the menu screen. In this case, when a user 
selects a certain menu E in the menu selection picture (operation-sides picture) 410 projected on 
the surface of projection 200, the color of the picture of that selected menu portion, etc. are 
changed like the above. 

The key selection detection part 180 of the above-mentioned embodiment and the actuated-valve- 
position detection part 380 of the above-mentioned modification are constituted so that key 
selection may be detected by finding the angle and distance of a position of a selection object key 
over a portable telephone body using a reflection of a laser beam, but they are not limited to this 
composition. For example, a motion of the finger 300 on the above-mentioned operation-sides 
picture 210,310,410 may be picturized, and the pattern recognition of the two-dimensional picture or 
three-dimensional picture may be carried out, and it may constitute so that a user actuated valve 
position may be detected based on the pattern recognition result. In this case, it may constitute so 
that the camera part 140 provided in the portable telephone 100 may be used also [ image pick-up / 
of a motion of the above-mentioned finger 300 ]. This camera part 140 is used for the image pick- 
up of a motion of the finger 300 which is carrying out input operation during execution in the input 
operation mode on the operation-sides picture 210,310,410 while it is used by photography of 
pictures, such as a person and scenery, during the usual operational mode execution. The key 
selection detection part 180 of the above-mentioned embodiment and the actuated-valve-position 
detection part 380 of the above-mentioned modification may be constituted so that the position of 
the finger 300 may be detected using a reflection of the ultrasonic wave instead of light like the 
above-mentioned laser. 

It may be made to project a picture which is displayed not only by the above-mentioned operation- 
sides picture 210,310,410 but by the described image display 119 by the described image projection 
part 130 in the above-mentioned embodiment and the above-mentioned modification. For example, 
it may be made to also project pictures relevant to the key selection performed on a keyboard 
picture, such as text and a game, pictures, such as this text, game, etc., — a portable telephone and 
the above — it will be easy to recognize visually if it projects between virtual keyboard pictures. 
[0040] 

The portable telephone 100 of the above-mentioned embodiment and the above-mentioned 
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modification may constitute the application program developed by the object oriented programming 
independent of a plat form so that execution is possible. And based on the instructions from an 
application program, it may be made to perform detection operation by the key selection detection 
part 1 80 of the above-mentioned embodiment, or the actuated-valve-position detection part 380 of 
the above-mentioned modification, and starting of the picture projection part 1 30 and the image 
projection operation by the picture projection part 130. As this application program, the application 
program etc. which were described with program language, such as JAVA (registered trademark), C, 
and C++, are mentioned. The execution environment of these application programs is built by 
middleware, such as VM of JAVA (registered trademark), and BREW (registered trademark). What it 
may be pre-installed in the portable telephone 100, was downloaded from the down load server via 
the portable telephone communication network as a communication network, and was saved and 
registered into the portable telephone 100 may be sufficient as this application program. 
[0041] 

Drawing 17 is a block diagram showing an example of the outline composition of a portable 
telephone which can execute an application program, and drawing 18 is an explanatory view showing 
an example of the software structure in the portable telephone. This portable telephone is provided 
with the application program execution control department 125 as an application program execution 
control means which manages the execution environment of an application program besides the 
above-mentioned main control part 1 1 1 grade. This application program execution control 
department 125 comprises a system bath, a CPU, RAM, etc., and operates according to the 
predetermined middleware for executing an application program. The application program execution 
control department 1 25, Central "program execution environment" is supported on the software 
structure of drawing 1 8 , Software used for the application program developed by object oriented 
programming, such as a class library, an execution environment management library, and application 
controlling, is provided, and the execution environment of an application program is managed. 
Here, the application program can use it now for class libraries, such as a function which is in the 
above-mentioned program execution environment via class library API (application interface), calling. 
The history of a call of class libraries, such as this function, is held until the virtual execution 
environment (virtual machine: VM) of an application program is completed. The execution 
environment management library in program execution environment can use it now for the telephone 
platform library in the below-mentioned telephone platform, calling via telephone platform API. 
Control of described image projection and key selection detection is performed by sending a control 
command to the picture projection part 130 or the key selection detection part 180 via the main 
control part 1 1 1 from the application program execution control department 1 25. It may constitute 
so that it may perform by sending a control command to the picture projection part 1 30 or the key 
selection detection part 1 80 directly from the application program execution control department 
125. 
[0042] 

awaiting in the portable telephone of the above-mentioned composition — business — during 
execution of application programs, such as an application program and a game, When performing 
image projection of a keyboard picture, and key selection detection, the predetermined function 
designator for image projection activate request instructions or a key selection detection request 
command is performed from an application program to an application program execution 
environment. The application program execution environment which received this function 
designator sends image projection activate request instructions and a key selection detection 
request command to the main control part 111. The main control part 1 1 1 which received these 
image projection activate request instructions and a key selection detection request command 
sends activate request instructions to the key selection detection part 1 80 while sending the 
control data for activate request instructions and image projection to the picture projection part 
130. 
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The picture projection part 1 30 is started according to the above-mentioned activate request 
instructions, and starts projection of the predetermined keyboard picture specified with the control 
data based on the control data for described image projection. The key selection detection part 180 
will be started according to the above-mentioned activate request instructions, and will be in the 
waiting state which can detect the key selection on the virtual keyboard picture projected by the 
picture projection part 130. 
[0043] 

The application program which performs starting, image projection operation, and key selection 
detection operation of the described image projection part 130 and the key selection detection part 
180, an incoming call notice picture is projected — awaiting — business — it may be the application 
program which may be an application program and which carried out and specialized in control of for 
example, other projector functions or the key selection detection on a virtual keyboard picture. 
[Brief Description of the Drawings] 
[0044] 

[Drawing 1] The control block diagram about the selection key picture change processing performed 

with the portable telephone concerning the embodiment of this invention. 

[Drawing 2] The block diagram showing an example of the outline composition of the portable 

telephone. 

[Drawing 3] The block diagram showing the example of 1 composition of the picture projection part 
of the portable telephone. 

[Drawing 4] The block diagram showing other examples of composition of the picture projection part. 
[Drawing 5] (a) And (b) is an internal configuration figure of the portable telephone in which the 
example of composition of further others of the picture projection part is shown. 
[Drawing 6] (a) is a block diagram showing the example of composition of further others of the 
picture projection part, (b) is a partial side view of the portable telephone provided with the picture 
projection part of drawing 5 . 

[Drawing 7] The perspective view of the portable telephone which has projected the keyboard 
picture. 

[Drawing 8] The block diagram showing the example of 1 composition of the key selection detection 
part of the portable telephone. 

[Drawing 9] The side view of the portable telephone which has projected the keyboard picture. 
[Drawing 10] The explanatory view showing the principle of key selection detection. 
[Drawing 11] The explanatory view showing the principle of the key selection detection concerning 
other examples of composition. 

[Drawing 1 2] The perspective view of the portable telephone for explaining signs that the color of the 
key image was changed by the selection key picture change processing. 

[Drawing 1 3] The perspective view of the portable telephone for explaining a situation when a user 
does key operation accidentally. 

[Drawing 14] The block diagram showing the outline composition of the example of 1 composition of 
the actuated-valve-position detection part 380 which the portable telephone concerning a 
modification has. 

[Drawing 15] The perspective view of the portable telephone for explaining signs that the color of the 

locus portion of the character which the user drew was changed in the modification. 

[Drawing 1 6] The perspective view of the portable telephone for explaining signs that the color of the 

menu which the user selected among menu screens was changed in another example. 

[Drawing 1 7] The block diagram showing an example of the outline composition of the portable 

telephone in which application program execution is possible. 

[Drawing 18] The explanatory view showing an example of the software structure in the portable 
telephone. 

[Explanations of letters or numerals] 
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[0045] 

100 Portable telephone 

1 1 1 Main control part 

113 Radio Communications Department 

119 Picture display part (liquid crystal display) 

130 Picture projection part 

1 60 Keystroke operation detecting part 

1 70 Key select data generation part 

1 80 Key selection detection part 

200 Surface of projection 

210 Keyboard picture 

300 Finger 

310 Handwritten input operation-sides picture 
370 Actuated-valve-position data generating part 
380 Actuated-valve-position detection part 
410 Menu selection picture 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 
[Claim 1] 

An image projection means for projecting an operation-sides picture which expresses virtually 
operation sides of operation equipment which a user operates, 

A display control means which performs display control of this image projection means, 

An actuated-valve-position detection means to detect a user actuated valve position on this 

operation-sides picture, 

It is the electronic device provided with a data processing means which performs data processing 
corresponding to a user actuated valve position detected by this actuated-valve-position detection 
means, 

An electronic device, wherein the above-mentioned display control means performs display control 
which changes at least a part of above-mentioned operation-sides picture according to a user 
actuated valve position detected by the above-mentioned actuated-valve-position detection means. 

[Claim 2] 

In an electronic device of Claim 1, 

An electronic device, wherein the above-mentioned display control is what changes an image region 
of a user actuated valve position detected by the above-mentioned actuated-valve-position 
detection means. 
[Claim 3] 

In Claim 1 or an electronic device of 2, 

An electronic device, wherein the above-mentioned operation-sides picture is a picture of a 
keyboard side where two or more keys for performing mutually different data processing to the 
above-mentioned data processing means were arranged. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 
[0044] 

[Drawing 1] The control block diagram about the selection key picture change processing performed with 
the portable telephone concerning the embodiment of this invention. 

[Drawing 2] The block diagram showing an example of the outline composition of the portable telephone. 
[Drawing 3] The block diagram showing the example of 1 composition of the picture projection part of the 
portable telephone. 

[Drawing 4] The block diagram showing other examples of composition of the picture projection part. 
[Drawing 5] (a) And (b) is an internal configuration figure of the portable telephone in which the example of 
composition of further others of the picture projection part is shown. 

[Drawing 6] (a) is a block diagram showing the example of composition of further others of the picture 
projection part, (b) is a partial side view of the portable telephone provided with the picture projection part 
of drawing 5 . 

[Drawing 7] The perspective view of the portable telephone which has projected the keyboard picture. 
[Drawing 8] The block diagram showing the example of 1 composition of the key selection detection part of 
the portable telephone. 

[Drawing 9] The side view of the portable telephone which has projected the keyboard picture. 
[Drawing 10] The explanatory view showing the principle of key selection detection. 
[Drawing 11] The explanatory view showing the principle of the key selection detection concerning other 
examples of composition. 

[Drawing 12] The perspective view of the portable telephone for explaining signs that the color of the key 
image was changed by the selection key picture change processing. 

[Drawing 13] The perspective view of the portable telephone for explaining a situation when a user does 
key operation accidentally. 

[Drawing 14] The block diagram showing the outline composition of the example of 1 composition of the 
actuated-valve-position detection part 380 which the portable telephone concerning a modification has. 
[Drawing 15] The perspective view of the portable telephone for explaining signs that the color of the locus 
portion of the character which the user drew was changed in the modification. 

[Drawing 1 6] The perspective view of the portable telephone for explaining signs that the color of the menu 
which the user selected among menu screens was changed in another example. 

[Drawing 1 7] The block diagram showing an example of the outline composition of the portable telephone in 
which application program execution is possible. 

[Drawing 18] The explanatory view showing an example of the software structure in the portable telephone. 
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[Drawing 8] 
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[Drawing 1 1] 
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[Drawing 1 2] 
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[Drawing 13] 
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[Drawing 14] 

5 8 



| | | | 



T 


7 


r 


ST 

j — I — «S£ 












! 




I 

« 

I 

as 


' • 










s 



[Drawing 1 5] 
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[Drawing 1 7] 
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[Drawing 18] 







i' 




. £3 




-A 



[Translation done.] 
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